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1 Interaction in the real world: Customizable physical interfaces for 77% 
13 interacting with conventional applications 

Saul Greenberg , Michael Boyle 

Proceedings of the 15th annual ACM symposium on User interface 

software and technology October 2002 

When using today's productivity applications, people rely heavily on 
graphical controls (GUI widgets) as the way to invoke application 
functions and to obtain feedback. Yet we all know that certain 
controls can be difficult or tedious to find and use. As an alternative, 
a customizable physical interface lets an end-user easily bind a 
modest number of physical controls to similar graphical 
counterparts. The user can then use the physical control to invoke 
the corresponding graphical contro ... 

2 Next-Gen Open Hypermedia, Part One: Links in the palm of your 77% 
12 hand: tangible hypermedia using augmented reality 

Patrick Sinclair , Kirk Martinez , David E. Millard , Mark J. Weal 
Proceedings of the thirteenth conference on Hypertext and hypermedia 
June 2002 

Contextualised Open Hypermedia can be used to provide added 
value to document collections or artefacts. However, transferring the 
underlying hyper structures into a users conceptual model is often a 
problem. Augmented reality provides a mechanism for presenting 
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these structures in a visual and tangible manner, translating the 
abstract action of combining contextual linkbases into physical 
gestures of real familiarity to users of the system. This paper 
examines the use of augmented reality in hyp ... 



3 Integrating paper and digital information on EnhancedDesk: a 77% 
13 method for realtime finger tracking on an augmented desk system 

ACM Transactions on Computer-Human Interaction (TOCHI) December 

2001 

Volume 8 Issue 4 

This article describes a design and implementation of an augmented 
desk system, named EnhancedDesk, which smoothly integrates 
paper and digital information on a desk. The system provides users 
an intelligent environment that automatically retrieves and displays 
digital information corresponding to the real objects (e.g., books) on 
the desk by using computer vision. The system also provides users 
direct manipulation of digital information by using the users' own 
hands and fingers for more natural ... 



4 Composable ad hoc location-based services for heterogeneous 77% 
12 mobile clients 

Todd D. Modes , Randy H. Katz 

Wireless Networks October 1999 

Volume 5 Issue 5 



5 Design and technology for Collaborage: collaborative collages of 77% 
12 information on physical walls 

Thomas P. Moran , Eric Saund , William Van Melle , Anuj U. Gujar , 

Kenneth P. Fishkin , Beverly L. Harrison 

Proceedings of the 12th annual ACM symposium on User interface 

software and technology November 1999 

A Collaborage is a collaborative collage of physically represented 
information on a surface that is connected with electronic 
information, such as a physical In/Out board connected to a 
people-locator database. The physical surface (board) contains items 
that are tracked by camera and computer vision technology. Events 
on the board trigger electronic services. This paper motivates this 
concept, presents three different applications, describes the system 
architecture and com ... 



6 TouchCounters: designing interactive electronic labels for physical 77% 
12 containers 

Paul Yarin , Hiroshi Ishii 

Proceeding of the CHI 99 conference on Human factors in computing 
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systems : the CHI is the limit: the CHI is the limit May 1999 

7 Touch-sensing input devices 77% 

13 Ken Hinckley , l^ike Sinclair 

Proceeding of the CHI 99 conference on Human factors in computing 
systems : the CHI is the limit: the CHI is the limit May 1999 



8 Beating the limitations of camera-monitor mediated telepresence 77% 
12 with extra eyes 

Kimiya Yamaashi , Jeremy R. Cooperstock , Tracy Narine , William 

Buxton 

Conference proceedings on Human factors in computing systems April 
1996 
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1 Multimodal human discourse: gesture and speech 80% 
12 Francis Quek , David l^cNeill , Robert Bryll , Susan Duncan , Xin-Feng 

l^a , Cemil Kirbas , Karl E. l^cCullough , Rashid Ansari 

ACM Transactions on Computer-Human Interaction (TOCHI) September 

2002 

Volume 9 Issue 3 

Gesture and speech combine to form a rich basis for human 
conversational interaction. To exploit these modalities in HCI, we 
need to understand the interplay between them and the way in 
which they support communication. We propose a framework for 
the gesture research done to date, and present our work on the 
cross-modal cues for discourse segmentation in free-form 
gesticulation accompanying speech in natural conversation as a new 
paradigm for such multimodal interaction. The basis for this integ ... 

2 Creating tangible interfaces by augmenting physical objects with 80% 
12 multimodal language 

David R. McGee , Philip R. Cohen 

Proceedings of the 6th international conference on Intelligent user 

interfaces January 2001 

Rasa is a tangible augmented reality environment that digitally 
enhances the existing paper-based command and control capability 
in a military command post. By observing and understanding the 
users' speech, pen, and touch-based multimodal language. Rasa 
computationally augments the physical objects on a command post 
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map, linking these items to digital representations of the same-for 
example, linking a paper map to the world and Post-ita notes to 
military units. Herein, we give a thorough ac ... 

3 Something fronn nothing: augmenting a paper-based work 80% 
3 practice via multimodal interaction 

David R. McGee , Philip R. Cohen , Lizhong Wu 

Proceedings of DARE 2000 on Designing augmented reality 

environments April 2000 

In this paper, we describe Rasa: an environment 
designed to augment, ratlier tlian replace, tlie worl< 
habits of its users. These work habits include drawing 
on Post-it™ notes using a symbolic language. 
Rasa observes and understands this language, 
assigning meaning simultaneously to objects in both 
the physical and virtual worlds. With Rasa, users 
rollout a paper map, register it, and move the 
augmented objects from one place to another on it. 
Once an object is augmented, users can m ... 



4 Session E: Interaction in mixed realities: A gesture processing 77% 
3 framework for multimodal Interaction in virtual reality 
Marc Erich Latoschik 

Proceedings of the 1st conference on Computer graphics, virtual reality 

and visualisation November 2001 

This article presents a gesture detection and analysis framework for 
modelling multimodal interactions. It is particulary designed for Its 
use in Virtual Reality (VR) applications and contains an abstraction 
layer for different sensor hardware. Using the framework, gestures 
are described by their characteristic spatio-temporal features which 
are on the lowest level calculated by simple predefined detector 
modules or nodes. These nodes can be connected by a data routing 
mechanism to perfor ... 



5 Expression constraints in multimodal human-computer interaction 77% 
13 Sandrine Robbe-Relter , Noelle Carbonell , Pierre Dauchy 

Proceedings of the 5th international conference on Intelligent user 
interfaces January 2000 

Thanks to recent scientific advances, it is now possible to design 
multimodal Interfaces allowing the use of speech and gestures on a 
touchscreen. However, present speech recognizers and natural 
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language Interpreters cannot yet process spontaneous speech 
accurately. These limitations make It necessary to Impose 
constraints on users' speech inputs. Thus, ergonomic studies are 
needed to provide user interface designers with efficient guidelines 
for the definition of usable speech constraints. ... 



6 Ten myths of multimodal interaction 77% 
13 Sharon Oviatt 

Communications of the ACM November 1999 

Volume 42 Issue 11 



7 Put that where? voice and gesture at the graphics interface 77% 

13 Mark Blllinghurst 

ACM SIGGRAPH Computer Graphics November 1998 
Volume 32 Issue 4 

A person stands in front of a large projection screen on which is 
shown a checl<ed floor. They say, "Make a table, " and a wooden 
table appears in the middle of the floor. "On the table, place a vase, " 
they gesture using a fist relative to palm of their other hand to 
show the relative location of the vase on the table. A vase appears 
at the correct location. "Next to the table place a chair. " A chair 
appears to the right of the table. "Rotate it like this, " while rotating 
their hand causes the cha ... 



8 Mutual disambiguation of recognition errors in a multimodel 77% 
12 architecture 
Sharon Oviatt 

Proceeding of the CHI 99 conference on Human factors in computing 
systems : the CHI is the limit: the CHI is the limit May 1999 



9 Embodiment In conversational interfaces: Rea 77% 
3 J- Cassell , T. Bickmore , M. Blllinghurst , L. Campbell , K. Chang , H. 
Vilhjalmsson , H. Van 

Proceeding of the CHI 99 conference on Human factors in computing 
systems : the CHI is the limit: the CHI is the limit May 1999 



10 QuickSet: multimodal interaction for distributed applications 77% 
3 Philip R. Cohen , Michael Johnston , David McGee , Sharon Oviatt , Jay 
Pittman , Ira Smith , Liang Chen , Josh Clow 

Proceedings of the fifth ACM international conference on Multimedia 
November 1997 



11 Iconic: speech and depictive gestures at the human-machine 77% 
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3 interface 

David B. Koons , Carlton J. Sparrell 

Proceedings of the CHI '94 conference companion on Human factors in 
computing systems April 1994 

12 Dialogue control In social interface agents 77% 

13 Kristin n R. Thorisson 

INTERACT '93 and CHI '93 conference companion on Human factors in 
computing systems April 1993 

13 Multimodal input for computer access and augmentative 77% 

13 communication 

Alice Smith , John Dunaway , Patrick Demasco , Denise PeischI 
Proceedings of the second annual ACM conference on Assistive 
technologies April 1996 

14 CyberBELT: multi-modal interaction with a multi-threaded 77% 
U documentary 

Joshua Bers , Sara Elo , Sherry Lassiter , David Tames 

Conference companion on Human factors in computing systems May 

1995 

15 An approach to natural gesture in virtual environments 77% 
U Alan Wexelblat 

ACM Transactions on Computer-Human Interaction (TOCHI) September 
1995 

Volume 2 Issue 3 

This article presents research—an experiment and the 
resulting prototype8imdash;on a method for treating gestural input 
so that it can be used for multimodal applications, such as 
interacting with virtual environments. This method involves the 
capture and use of natural , empty-hand gestures that are made 
during conventional descriptive utterances. Users are allowed to 
gesture in a normal continuous manner, rather than being restricted 
to a small set of discrete gestural commands as in ... 
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